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Abstract 

This paper evaluates compression of AVHRR imagery operating in a lossless 
or nearly-lossless mode. Several practical issues are analyzed including: 
variability of compression over time and among channels, rate-smoothing 
buffer size, multi-spectral preprocessing of data, day/night handling, and 
impact on key operational data applications. This analysis is based on a 
DPCM algorithm employing the Universal Noiseless Coder, which is a 
candidate for inclusion in many future remote sensing systems. It is shown 
that compression rates of about 2:1 (daytime) can be achieved with modest 
buffer sizes (< 2.5 Mbytes) and a relatively simple multi-spectral preprocessing 
step. 



Introduction 

Incorporation of compression into a real-time remote sensing system adds a number of 
complications. Lossless compression, desired by many users, necessarily results in a variable 
rate output. A rate smoothing buffer is thus required to interface to systems which require a 
fixed rate input such as real-time downlinks and magnetic tape mass storage. Also, since the 
possibility of buffer overflow cannot usually be eliminated, some means must be incorporated 
to reduce the rate below that achieved by lossless compression in such situations. Coding 
delay may also be an issue for real-time downlinks depending on the size of the buffer. 

Martin Marietta Astro-Space Division has developed a test-bed consisting of both hardware 
and software to investigate such issues. The test-bed consists of: (1) a wide variety of 
compression algorithms (including both industry standard algorithms such as the Universal 
Noiseless Coder, the Joint Photographic Experts’ Group discrete cosine transform algorithm 
and internally developed algorithms); (2) system modeling software such as rate smoothing 
buffers; and (3) diagnostic software to characterize compression algorithm performance and 
develop appropriate metrics. Most of the compression algorithms are implemented in a 
workstation environment. A number of algorithms are implemented on a real-time 
programmable signal processor. In this study, the test-bed was applied to investigate lossless 
compression of the Advanced Very High Resolution Radiometer (AVHRR) which flies on the 
TIROS series of low-altitude weather satellites. 


AVHRR Data Set 

A data set consisting of real-time AVHRR data acquired from the NOAA 1 1 and 12 satellites 
was assembled. The data were received at a High Resolution Picture Transmission (HRPT) 
Receiving Station which is part of the Advanced Remote Sensing Laboratory at Martin 
Marietta Astro-Space Division in Princeton, New Jersey. Both day and night passes were 
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assembled consisting of ~60 minutes of daytime data and ~57 minutes of nighttime data 
acquired from the ground station located in Princeton. A typical pass duration was 8-10 
minutes. The data set covers a variety of regions and scene complexities, including ocean and 
land over latitudes ranging from 25°N to 55°N. Total data set size was about 860 Mbytes — 
somewhat greater than the data from one complete orbit. 

The uncalibrated HRPT data were used in the analyses that follow. These data are for five 
bands (two in the visible/near infrared, one mid-wave infrared and two long-wave infrared) 
and have a spatial resolution of about 1 . i km at nadir for -2,048 samples per scan line. Each 
sample is quantized to 10-bits. The HRPT data stream also contains sensor calibration 
samples, spacecraft telemetry data, frame synchronization and other miscellaneous headers, 
and data from lower rate sensors. Compression of these other data was not investigated. 
Total data rate of the HRPT stream is 666 kbits/s. 

Compression Approach 

A Differential Pulse Code Modulation (DPCM) coder followed by an entropy coder was used 
as illustrated in Figure 1. Both one- and two-dimensional (1-D and 2-D) predictors were 
tested. A simple three-point, 1-D predictor was used for most of the results reported here. 1 - 
D predictors minimize front-end buffering and simplify error propagation control. Entropy 
coding was based on the Universal Noiseless Coder (UNC) described by Rice (1991). The 
UNC was selected for several reasons: competitive performance when compared to other 
entropy coders for the type of data used in this study; anticipated availability in high-speed, 
rad-hard chips; and its inclusion in the Consultative Committee for Space Data Systems 
(CCSDS) standard for Advanced Orbiting Systems, Networks and Data Links (Yeh, et al, 
1992). 



Note : ' DPCM predictor may alternately feedback to spectral preprocessor as 
described in text which would modify diagram (not shown) 


Figure 1 DPCM Model Block Diagram 

The UNC implementation employed eight of the alternative Rice coders 'P]_o through 'P|,6 
plus the default coder 4 ^ 3 . In Rice’s nomenclature this translates to a coder with values X = 1 
and N = 8. No additional coding of the coder identifier was performed. We experimented 
with a variety of block sizes (J in Rice’s nomenclature) and determined that J = 16 or J=32 
were near optimum for most cases. The starting values for the DPCM predictor were 
provided only once per scan line. 

When operated in a lossless mode the quantizer of Figure 1 is the identity function. A 
uniform quantizer was used for lossy operation, as described later. 
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Although most of the experiments described here were performed on Sun SPARC-2 and 
SPARC- 10 workstations, these algorithms have also been implemented on a real-time 
programmable signal processor developed by Martin Marietta built around the Texas 
Instrument TMS-320C30 chips. Rates in excess 1.5 Mpixels/s have been demonstrated on a 
four-processor version. Such a system may be an alternative to firmware solutions for 
moderate rate applications desiring flexibility and reprogramability. 

Some special procedures were added for nighttime data. While there is essentially no 
information in channels 1 and 2 at night for normal conditions (they measure reflected solar 
radiation), it is possible that such data might be of use for unusual circumstances. For this 
reason the channels were not completely eliminated in the final formatted product. Rather, at 
night the signal level which consists of the zero level plus random noise was replaced by a 
fixed value (in this case zero) when the signal is within some range determined by the 
expected noise level. This function could be implemented outside the UNC chip. This 
method provides a very high compression (»20: 1) for these channels but would still acquire 
rare special events at night with negligible impact to the overall performance. 

Multi-spectral Preprocessing 

It has long been recognized that spectral correlations among sensor bands can be used to 
further improve compression of multispectral data. However, since this decorrelation adds to 
the complexity of the system, its marginal benefit must be carefully weighed. In the case of 
the AVHRR, this improvement has been found to be small, but perhaps significant in some 
applications. Miettienen (1992) using a Discrete Cosine Transform (DCT) spatial compressor 
preceded by a Karhunen-Loueve spectral transform (KLT) found an 18% reduction in rate 
compared to spatial compression only for a fixed mean squared error (mse) at moderate 
compression ratios (8:1 to 15:1) but at low compression ratios and low mse (mse < 1 digital 
numbers per band and compression ratio < 6:1), the incremental benefit was less than 8%. As 
lossless performance is approached, this benefit is further reduced. 

Among AVHRR bands, numbers 4 and 5 have the highest correlation (in excess of 95%). 
Both measure thermally emitted radiation in the 10-12 pm window region with most of the 
brightness temperature differences (A Tb almost always less than 2 K) arising from small 
differences in water vapor absorption (for scenes viewing the surface). Thin cirrus (ice) 
clouds have been shown to likewise result in a small but significant signature in ATb- The 
compression of each channel individually was compared to sending bands 1 through 4 plus 
the difference of bands 4-5. For lossless compression, a reduction in data rate of 5.5% was 
achieved when averaged over all bands (reduction from 5.25 to 4.96 bits per pixel per band, 
bpppb). 

The final algorithm also employed the differences of bands 1 and 2 which reduced the rate 
another few percent. No spectral preprocessing was applied to band 3 (~3.7 pm) which 
responds to both thermally emitted radiation and reflected solar radiation during the day and 
shows only modest correlation with the other bands. This is probably due to a combination of 
the more complex phenomenology and the excess sensor noise often experienced by this 
channel. 

An additional modification must be made to allow a lossy mode. One possibility is to 
include the spectral preprocessor in the DPCM feedback loop (see Figure 2a). While not 
inherently difficult to implement it does add to the complexity of the spectral and spatial 
compressor interface. An alternative is to send both the difference and sum of the bands 
(Figure 2b). As any errors introduced by the quantization step are now orthogonal, no 
feedback is necessary. The reader will undoubtedly recognize this as the degenerate ca.se of 
the KLT for two bands (without the scaling) — the only KLT which is data independent. 
Figure 2c illustrates a five-band orthogonal spectral preprocessor. As long as the KLT 
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transform vectors are prestored (i.e., calculated on the ground) and not calculated in real-time, 
this presents only a modest computational burden. 

Thus, it has been found that an optimal five-band spectral transform (i.e., KLT) is not 
necessary to secure most of the advantage from spectral correlations for a multispectral 
compressor. Operating on differences between bands 1 & 2 and 4 & 5 has the added benefit 
that several of the key applications of AVHRR data employ these channel differences in a 
rather direct way (e.g., .sea surface temperature and normalized difference vegetation index). 
This naturally leads to methods for optimizing the compression algorithm for user processing. 

Rate-Smoothing Buffer Sizing 

A model was developed which emulated the system of Figure 1. The following parameters are 
specified for the rate smoothing buffer: buffer size (in bytes), initial buffer state (percent 
full), and fixed output rate. The day and night AVHRR data were then separately processed 
by the model. Statistics were kept for the fraction of time the rate smoothing buffer was 
maintained in various states of fullness. 


Ch #4 
Ch #5 


Ch #4 


Ch #5 


^ to DPCM coder A 



to DPCM coder B 

. feedback from coder A predictor (Ch#4) 
(similarly for channels 1&2) 

(a) Spectral DPCM Preprocessor 



to DPCM coder A 

to DPCM coder B 
(similarly for channels 1&2) 


(b) Orthogonal Spectral Preprocessor 



to DPCM 
coders A-E 


(c) Five-band Orthogonal Spectral Preprocessor 
Figure 2 Alternative Spectral Preprocessors 
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Figure 3 gives an example of the variable compression ratio (averaged over single AVHRR 
scan lines) versus scan line number for a typical pass. The compression variability , ranges 
from 6.6 bpp to 4.8 bpp. The very low rates are communication drop-outs experienced by the 
receiving station. 

Figure 4a shows a histogram of buffer state for the daytime date set with a 2.5 Mbyte buffer 
and a 4.9 bpp fixed rate output. The buffer was in an overflow state approximately 1 % of the 
time. These conditions can be handled by one of several approaches discussed in the next 
section. Figure 4b shows a similar histogram for the nighttime data set. 



Figure 3 Line-averaged Lossless Compression Rate for a Typical AVHRR Pass 



(a) Daytime Data Set _ (b) Nighttime Data Set 

Figure 4 Buffer State Histograms 
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Next the fixed out rate was varied with a fixed buffer size. Figure 5a plots the fraction of data 
overflowed in the buffer versus the fixed output rate for buffer sizes of 1 and 2 Mbytes. 
These results suggest that a buffer size of 2 Mbytes corresponding to about 3.5 minutes of 
data is sufficient to operate losslessly all the time at a fixed output rate of 5. 1 bpppb. Similar 
calculations with nighttime data (Figure 5b) indicate that a rate of 3 bpppb (averaged over 5 
bands) can be achieved with a 5 Mbyte buffer. A smaller buffer only increases the amount of 
buffer overflow by a small amount («l% for 1.0 Mbyte buffer). 



(a) Daytime (b) Nighttime 

Figure 5 Buffer Overflow versus Output Rate for Various Buffer Sizes 
Graceful Degradation Mode 

Rice (1991) describes several methods for adapting the UNC to a lossless mode. The leading 
candidates described are truncation at the edge of the scan and progressive elimination of low 
order bits. The former method is reasonable for planetary missions where a camera is 
centered on a target of interest (typical of planetary missions for example). It is less 
reasonable for a system such as the AVHRR where global coverage and continual monitoring 
are desired. In the second method, the elimination of high order bits can be facilitated by an 
appropriate ordering of the UNC output stream. This method provides all the data and the 
loss can be selectively applied (for example to lower priority regions). A number of 
implementation variants are also described such as a zig-zag ordering method which may 
offer an advantage for some applications. 

For this paper, a third approach is used which provides the rate control feedback through the 
quantizer. A uniform quantizer is used which has been shown to provide nearly optimum 
performance — in terms of its rate distortion function — for a scalar quantizing system using 
entropy coding of a memoryless source (Farvardin and Modestino, 1984). 

Some trades of rate versus distortion for the uniform quantizer are shown in Figure 6. The 
rate is reduced from 5.5 bpppb lossless to 3.8 bpppb with an mse of -0.6 DN^ (digital 
numbers). Thus, significant control of output rate can be achieved with very modest errors 
introduced to the data. This distortion plateaus near an mse of 0.5 DN^ due to the 10-bit 
quantization of the data input to the quantizer. By maintaining more bits precision in the 
multi-pixel predictor (or preferably in the original sensor data), rounding problems with the 
uniform quantizer can be minimized. The rate-distortion curve would then exhibit a more 
gradual degradation. 
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Figure 6 Rate Distortion Function for Uniform Scalar Quantizer 

The proposed rate control method consists of determining a parameter T, the lossless/lossy 
buffer fullness threshold, and a function r(S), the quantizer feedback. As long as the buffer 
state of fullness S < T, the system operates in a lossless mode. When S > T, the uniform 
quantizer is supplied with a divisor determined by r(S). Further experiments are required to 
determine the optimal T and r(S). It appears that a linear function will be adequate. 

Discussion 

A model has been developed for evaluating lossless compression performance using the 
Universal Noiseless Coder and applied to the AVHRR. A variety of system parameters can 
be traded using this model such as buffer size, fixed output rate, etc. It has been determined 
that a strictly one-dimensional compressor using a 3 point predictor can achieve compression 
from the original 10-bit AVHRR data to ~5 bits per pixel per band for daytime and ~3 bits per 
pixel per band for nighttime with buffer sizes less than 2 Mbyte. The results summarized in 
Table 1 indicate that even for the nearly-lossless mode, that maximum errors of < 1 DN. The 
corresponding mean square errors would be «1. 


Table 1 Lossless and Nearly-Lossless Compression Summary 


Mode 

Rate (bpp) 

Buffer (MB) 

Lossy Fraction* 

Max. error (DN)** 

Day: 



<1% 

1 

Lossless 

5.1 

2.0 

Nearly-lossless 

4.9 

1.0 

~4% 

1 

Night: 



<1% 

1 

Lossless 

3.0 

2.0 

Nearly-lossless 

2.8 

1.0 

~4% 

1 

Notes: * Fraction of time spent is lossy mode 

** Estimated maximum error during lossy mode, mse < 0.5 


The 1-D compressor described has the advantage that any bit stream errors cannot propagate 
beyond the line in which they occur. The maximum coding delay of 3.5 minutes is not 
expected to be significant for most situations. A simple sum/difference spectral preprocessor 
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applied to channels 1&2 and 4&5 respectively was shown to provide a small but potentially 
useful reduction in rate (6-8%) when compared with compressing each channel 
indep>endently. 

While the system parameters above can provide lossless performance the vast majority of the 
time, buffer overflow might still occur. A feedback system to a uniform quantizer was 
recommended and examples of the rate distortion function were given. Should a fixed output 
rate near the average lossless rate be selected, this could provide a fallback mode for rare 
circumstances when the buffer overflows. Since errors are small — less than the inherent 
noise level of the sensor — the impact on data quality would be very small. Should a rate 
below the lossless average rate be desired, the rate distortion function suggests that mse < 1 
DN^ can be achieved with rates up to 2 bpp below the lossless rate. 

While silicon implementations of the UNC are available (Yeh, et al, 1992), additional support 
circuitry would be required in any event to perform the spectral and spatial predictions and to 
implement the quantizer. An alternative is to employ programmable signal processors. This 
adds considerably to the flexibility of the compressor. Minor and possibly major 
modifications to the algorithm could be made even during a mission. The UNC has been 
tested in such a system at Martin Marietta. The programmable signal processor uses four 
Texas Instrument TMS320C30 processor supplemented by custom interface chips to enhance 
interprocessor communications. The UNC algorithm, using a somewhat simpler predictor 
than described here, has been benchmarked at rates in excess of 1.5 Mpixels/s on this system. 
This is much greater than the ~60 kpixels/s rate at which the AVHRR operates. 

Future work will expand the model in a number of ways. The graceful degradation mode will 
be integrated with the overall model. Ability to analyze the impact of bit stream errors will 
also be incorporated. Furthermore, radiometricully critical AVHRR applications such as Sea 
Surface Temperature (SST) and Normalized Difference Vegetation Index (NDVI) will be 
investigated. Additionally, greater quantities of data will be tested and other multispectral 
sensors will be considered. 
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